System and method for detecting undesirable and potentially harmful online behavior

ABSTRACT

Embodiments include computer-implemented methods and systems for detecting undesirable and potentially harmful online behavior. The embodiments described and claimed could also be applied to detecting any other type of online behavior to be detected, but the descriptions focuses on detecting online violence. More particularly, the embodiments disclosed relate to detecting online violence using symbolic methods of natural language processing (NLP) that utilize and govern the usage of: 1) syntactic parser for analyzing grammatical context of the input text data, 2) unsupervised learning methods for improving selected aspects of the system and adjusting the system to new data sources and guidelines, and 3) statistical classifiers for resolving specific well-defined sub-tasks, in which statistical approaches surpass the symbolic methods.

RELATED APPLICATIONS

The present application is a continuation of, relates to, and claims thebenefit of priority to U.S. patent application Ser. No. 17/191,429 filedon Mar. 3, 2021, which claims benefit of priority of U.S. patentapplication Ser. No. 16/289,881 filed on Mar. 1, 2019, which claimsbenefit of priority to U.S. Provisional Patent Application No.62/638,073, filed Mar. 3, 2018, which are hereby incorporated byreference herein in their entirety for all purposes as if fully setforth herein.

FIELD OF THE INVENTION

Inventions disclosed and claimed herein are in the field of naturallanguage processing (NLP).

BACKGROUND

Online violence can be broadly defined as any form of abusing,harassing, bullying or exploiting other people, using electronic means.Some communication phenomena such as hate speech, toxic speech orabusive language overlap with online violence to some extent, whereasthe other phenomena like cyberbullying or sexual harassment are entirelyincluded into online violence.

Online violence puts emphasis on harming other people. For example,using the word “Peking” as an intensifier to stress out the positiveemotions towards other person, e.g. “you are Peking awesome”, is notonline violence.

A number of methods exist for detecting online violence, its subclassesor similar phenomena. Most of them utilize statistical learningtechniques such as machine/deep learning, where a classification modelis trained on a set of labeled examples. FIG. 2 is a block diagram ofprior art online violence detection system 200. The input is a text,which is processed by Online Violence Detection System 202 consisting ofMachine Learning Techniques 204 trained on Manually Annotated Corpus206. Online Violence Detection System 202 produces Online ViolenceDetection System Output 208 which is then used by OtherSystems/Applications 150.

Although the learning capabilities of statistical learning techniquesare impressive, especially the ones utilizing deep artificial neuralnetworks, this approach is lumbered with many important drawbacks,including (but not limited to):

1. It requires a large set of annotated examples for each category.Therefore, the systems built with this approach usually offer a verylimited number of categories or even only a binary classification (e.g.toxic/non-toxic). Furthermore, the most dangerous phenomena are usuallyvery rare so there is a problem in getting large enough labeled datasetto train an accurate classifier.2. It tends to over-focus on certain keywords and expressions that oftenco-occur with online violence such as vulgarisms. For example, manystatistical systems have problems recognizing even the simplest forms ofnegation such as “you are not an idiot” or recognizing non-abusivecontext in utterances like “rapists should spend rest of their life inprison.”3. It is vulnerable to adversarial attacks. Adversarial examples areinputs to machine learning models that an attacker has intentionallydesigned to cause the model to make a mistake. In terms of onlineviolence detection, it can be: word changes (e.g. inserting a typo),word-boundary changes (e.g. inserting or removing whitespaces), or wordappending (e.g. inserting positive non-hateful words).4. It is a “blackbox” as it is impossible to say why the system madecertain decision. One cannot track the decision-making process andcannot change anything within the model other than by retraining it on adifferent dataset or changing the learning algorithm or thearchitecture. It has a number of negative consequences, but the mostobvious are significant difficulties in developing and maintaining suchsystem.5. It strongly depends on the annotation quality and the data sourceused for training. A statistical model trained on one data source (e.g.Twitter) shows a significant drop in accuracy when used on other datasource (e.g. Reddit). The same applies to the annotation quality. If amodel was trained using given annotation guidelines, it will not workwell for other guidelines. Furthermore, if the annotation containsmistakes, it will affect the model's performance.

Statistical models achieve low precision that effectively excludes thesystems using such models from being used as proactive preventioninstead of post factum moderation within a human-in-the-loop model.According to other limitations, such systems are not effective indetecting rare online violence phenomena and can be easily cheated orspoofed with adversarial examples. The latter affects the recall of suchsystems when applied to real online communities, especially if the usersknow that there is an online violence detection system monitoring theircommunication. Furthermore, such systems cannot be customized orrelatively quickly adjusted to different services, data sources andguidelines, which is an important market need.

On the other hand, symbolic systems demonstrate even more severedrawbacks, which exclude them from the mainstream in both academic andindustry applications. Due to the lack of learning capabilities,symbolic systems mostly rely on hand-crafted rules and knowledge bases.The process of building rules and knowledge bases is tedious, expensiveand usually requires a number of well-educated specialists representinga range of disciplines such as computer science, linguistics,psychology, and so on. Furthermore, as the complexity of such systemincreases, it becomes very hard, and sometimes nearly impossible, todevelop and maintain these systems.

It would be desirable to have an online violence detection system thatcombines the advantages of symbolic and statistical approaches, greatlyreducing or even eliminating the individual drawbacks of each of them.It would be desirable to have an online violence detection system thatis able to detect and name a range of online violence phenomena,including the rare dangerous phenomena, with both high precision andrecall. High precision would be an enabling factor for changing the wayin which online services handle the problem of online violence—from postfactum moderation within the human-in-the-loop model to automaticprevention allowing the blocking of dangerous content before the damageis done. It would be desirable to have an online violence detectionsystem that deeply analyzes the context in order to precisely detectcomplex and non-obvious online violence cases, and to prevent raisingtoo many false alarms. It would be desirable to have a method forbuilding and developing such online violence detection system that donot require large sets of annotated data, or even to enable such onlineviolence detection system to learn directly from unlabeled datasets.Such a method should allow the labeled and unlabeled datasets to bedrawn upon in order to make the system better, but should not make thesystem dependent on those datasets. It would be desirable to have aneffective method for incorporating the knowledge of experts such aspsychologists, sociologists, pedagogues, psychiatrists into the system,and for customizing and adjusting the system to new data sources,community guidelines, and other client's needs. It would be desirable tohave a method for developing and maintaining such system that allows toprecisely find the root causes of errors and problems, to plan thedevelopment short- and long-term, and to build the development teamsthat could work on different aspects of the system independently.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an online violence detection systemenvironment according to an embodiment.

FIG. 2 is a block diagram of a prior art online violence detectionsystem.

FIG. 3 is a block diagram of an online violence detection systemaccording to an embodiment.

FIG. 4 is a block diagram of text preparation according to anembodiment.

FIG. 5 is a block diagram of normalization (part of text preparation)according to an embodiment.

FIG. 6 is a block diagram of correction (part of text preparation)according to an embodiment.

FIG. 7 is a block diagram of transformation (part of text preparation)according to an embodiment.

FIGS. 8-9 are diagrams illustrating the process of text preparationaccording to an embodiment.

FIG. 10 is a block diagram of detection engine according to anembodiment.

FIG. 11 is a block diagram of violence towards second person detection(part of detection engine) according to an embodiment.

FIG. 12 is a diagram illustrating the division of online violencephenomena into sub-classes according to an embodiment.

FIG. 13 is a block diagram of transformations to second person (part ofdetection engine) according to an embodiment.

FIG. 14 is a block diagram of other violence detection (part ofdetection engine) according to an embodiment

FIG. 15 is a block diagram of general architecture of online violencedetection modules according to an embodiment.

FIG. 16 is a block diagram of symbolic processing component according toan embodiment.

FIG. 17 is a diagram illustrating syntactic parser output according toan embodiment.

FIGS. 18-19 are diagrams illustrating the operation of online violencedetection modules according to an embodiment.

DETAILED DESCRIPTION

The present invention relates to computer-implemented methods andsystems for detecting undesirable and potentially harmful onlinebehavior. The embodiments described and claimed could also be applied todetecting any other type of online behavior to be detected, but thedescriptions focus on detecting online violence. More particularly, theembodiments disclosed relate to detecting online violence using symbolicmethods of natural language processing (NLP) that utilize and govern theusage of: 1) syntactic parser for analyzing grammatical context of theinput text data, 2) unsupervised learning methods for improving selectedaspects of the system and adjusting the system to new data sources andguidelines, and 3) statistical classifiers for resolving specificwell-defined sub-tasks, in which statistical approaches surpass thesymbolic methods. The online violence detection system comprises ahierarchical modular architecture, utilizing contextual models to detectspecific well-defined online violence phenomena. The contextual modelsare built using the above-mentioned methods and organized into modulesand submodules of increasing complexity representing responding typesand subtypes of online violence.

Embodiments of inventions disclosed herein include improvements oncurrent NLP systems and methods for an automatic detection of onlineviolence by processing input that consists of text from different typesof context. The input text can be obtained from any source of electronicdata that could serve as a source of text input to the NLP system,including (but not limited to): plain text, text files, word processingdocuments, CSV files, XML, files.

FIG. 1 is a block diagram of an online violence detection systemenvironment 100 according to an embodiment. Online Violence DetectionSystem 110 accepts text as input. Text can include electronic data frommany sources, such as the Internet, physical media (e.g. hard disc), anetwork connected database, etc. Online Violence Detection System 110includes multiple System Databases 110A, Dictionary 540 databases andRule Libraries 550 databases, and multiple System Processors 110B,Syntactic Parser 520 processors and Statistical Classifiers 530processors. Databases 110A, 540, 550 and processors 110B, 520, 530 canbe located anywhere that is accessible to a connected network 120, whichis typically the Internet. Databases 110A, 540, 550 and processors 110B,520, 530 can also be distributed geographically in the known manner.Data Sources 130 include any source of electronic data that could serveas a source of text input to Online Violence Detection System 110.

Other Systems/Applications 150 are systems, including commercial andnon-commercial systems, and associated software applications that havethe capability to access and use the output of Online Violence DetectionSystem 110 through one or more application programming interface (APIs)as further described below. For example, Other Systems/Applications 150can include:

-   -   an online application offering its users a chatting option that        is monitored for online violence and proactively moderated by        blocking messages containing online violence before they reach        the receiver (online violence prevention),    -   a mobile application for parents that monitors their children's        online communication and alerts when children is exposed to        online violence.

End Users 140 includes individuals who might use OtherSystems/Applications 150 through one or more of End User Devices 140A.End User Devices 140A include without limitation personal computers,smart phones, tablet computers, and so on. In some embodiments, EndUsers 140 access Online Violence Detection System 110 directly throughone or more APIs presented by Online Violence Detection System 110.

Online violence (or cyber violence) is any form of abusing, harassing,bullying or exploiting other people, using electronic means. In otherwords, the term “online violence” used in this document should beunderstood in the broadest sense—as any activity or behavior of onlinecommunication that can harm or lead to harming other people, with thestrong emphasis on (but not limited to) protecting an interlocutor.Phenomena such as personal attacks, sexual harassments, cyberbullying,threatening, blackmailing naturally fall to this category. For reasonsof clarity, also some non-obvious phenomena are included into thiscategory, such as: sexting, sex trafficking, racism, sexism, violenceagainst minorities, persuading self-harm and violence against others.According to the definition, each of these phenomena can be seen as anexample of abusing, harassing, bullying or exploiting other people,using electronic means. For example, sexting can be seen as an exampleof exploitation or harassing if one knows that the receiver of themessage is a child that should not be presented with such content.Another special emphasis is put on protecting children, and thereforeonline violence covers any violent, harming or exploiting phenomenatargeted against children, including: child grooming, childexploitation, pedophile attacks, extortion or coaxing of impropriateitems or actions (e.g. nude pictures), and so on. Additionally,declaration of self-harming is included to online violence as anexception since it does not fulfill the definition. Although it is nottargeted against other people, it is a very dangerous phenomenon thatshould be detected in order to prevent putting words into action.

The system and method comprise a novel approach to online violencedetection that combines statistical methods (including machine and deeplearning models) under the strict government of the symbolic processing,strongly relying on grammar structure. The system and method can becharacterized by two important advantages in comparison to the state ofthe art approaches: high precision and broad division into diverseonline violence phenomena.

High precision is an enabling factor for the detection system to be usedfor active prevention rather than post factum moderation. Currentsystems achieve low precision which results in a large number of falsealarms. Taking an automatic action against a user detected with lowprecision system is very risky because such action can be a falseaccusation that effectively leads to the user's dissatisfaction or evenresignation from using the service. Therefore, current system can beused as an aid for human moderation, not as an autonomous preventionsystem.

Broad division allows the assignment of a specific action to a specificdetected phenomenon. The system and method allows the building andexecution of narrow specialized detection models with a minimal amount(or even none) of training data. For example, one can set up differentreactions for the detection of personal attacks, blackmailing and childgrooming. Furthermore, some narrow models such as detection of nudepicture extortion can be used to educate the potential victim, e.g. byinforming that it is okay to tell parents and not fulfill theserequests.

FIG. 3 is a block diagram illustrating the high-level architecture ofthe inventions according to an embodiment. Input text is received byText Preparation 112. Text Preparation 112 is responsible for preparingthe input text to be further processed by Detection Engine 114.Detection Engine 114 is responsible for determining if the input textcontains online violence and—if so, what is the category of detectedonline violence. The output of the Detection Engine 114 is referred tohere as System Output API 116, and can be operated on by a variety ofdownstream software applications or systems as described in thisdocument, referred to here as Other Systems/Applications 150. Althoughthe output of the system is referred as Application ProgrammingInterface (API) in System Output API 116, any reasonable alternativesare possible.

Text Preparation

Detection Engine 114 requires a normalized and corrected text tooperate. The goal of Text Preparation 112 is to perform a broad set oftext manipulation operations in order to convert the input text into thetext of the form required by Detection Engine 114. Text Preparation 112is the only of the two system's components that allows the manipulationof the text string by changing, adding or removing the sub-strings. Theoriginal text string is preserved as a separated entity “text_orig.”

FIG. 4 is a block diagram showing the process of Text Preparation 112according to an embodiment. The input of Text Preparation 112 consistsof plain text. The process of Text Preparation 112 comprises threeconsecutive modules: Normalization 302, Correction 304, andTransformation 306. The standard process utilizes one time pass.However, it is possible to perform subsequent passes if needed. EveryText Preparation 112 module has an access to all of the system's assets,features and functions such as syntactic parser, statisticalclassifiers, dictionaries and libraries. Any of these resources can beused in any module and submodule. Normalization 302 comprises theprocess of adjusting an input text into a predefined standard format.

FIG. 5 is a block diagram providing detailed division of Normalization302 into a set of submodules realizing specific tasks.

Character Encoding 302A is responsible for converting all characters ininput text to a default encoding, including characters and charactercodes in any known form (e.g. “\u2018”, “0x2018”). Redundant Repetitions302B deletes a variety of well-defined sub-string repetitions in inputtext, including (but not limited to):

-   -   repetitions of letters (more than two to two as there are no        English words with more than two occurrence of the same letter),    -   repetitions of words (with exception for meaningful repetitions        such as proper nouns),    -   repetitions of whole utterances,    -   repetitions of punctuation marks.

Tags/Markers/Code Snippets 302C deletes or changes to acceptable wordsand characters a variety of usage of markups and formal (not natural)languages, including (but not limited to): HTML tags, BBCode tags, CSSstyling, source-related tags, any known programming language codesnippets.

Emoticons 302D deletes or changes detected emoticons (including variousstyles such as western style, kaomoji) into acceptable words andcharacters. Each emoticon is changed into descriptive expression from aninput text language, e.g. “:-(” can be changed to “i am sad.”sub-string.

Contractions 302E expands shortened versions of the written forms of awords into their full normalized version, e.g. “you're” into “you are.”It also covers unambiguous incorrect contractions such as “youre” into“you are,” but not “hell” as it can be “he will” as well as the word“hell.”

Whitespaces 302F normalizes any occurrence of whitespaces, including(but not limited to): spaces, tabulations, next lines. The normalizationcan be different for different occurrences of the same whitespace.

Segmentation 302G is responsible for controlling the sentence division.It detects and inserts omitted full stops, and handles scattered textpieces (e.g. using certain whitespaces or tags introducing tabulationsor new lines). It also secures the usage of dots in abbreviations.

Entity Recognition 302H covers the recognition of a variety ofwell-defined entities, including (but not limited to):

-   -   numbers,    -   series of numbers,    -   mathematic equations,    -   date/time information,    -   publication/article details (e.g. publication date, references),    -   identification numbers (IDs),    -   telephone/fax numbers,    -   IP addresses,    -   email addresses,    -   Uniform Resource Locators (URLs),    -   geographic coordinates,    -   money amounts,    -   filenames,    -   domain- and source-specific entities (e.g. Reddit username and        subreddit conventions).

The recognized entities are changed into a normalized form that isfurther handled by Detection Engine 114 and other Text Preparation 112modules.

Other Normalization 3021 covers every other normalization that does notfall into the above-mentioned modules.

Correction 304 (as described below with reference to FIG. 6) comprisesthe process of revising the normalized text in order to removemisspellings and any other errors and obstacles that may impede the workof Detection Engine 114. The process of Correction 304 (its everysubmodule) utilizes two major approaches:

-   -   dictionary-based—using various algorithms for calculating        “distances” (or similarity metrics) between misspelled words and        correct dictionary entries; it may result in a replacement or in        generating a list of possible candidates;    -   context-based—using various algorithms for selecting a proper        candidate by establishing what is required within a given        context; it may find a replacement by itself or use a list of        candidates for other module or method, including the usage of        dictionary-based correction; examples of context-based        correction: “its a miracle” into “it is a miracle”, “you are eal        stupid” into “you are really stupid.”

FIG. 6 is a block diagram providing detailed division of Correction 304into a set of submodules realizing specific tasks.

Common Typos (Statistical) 304A replaces incorrectly written words withtheir correct version based on statistical data, e.g. “accomodation”into “accommodation.” The statistical data can come form any source,including (but not limited to):

-   -   existing databases of common typos,    -   dedicated submodules (internal or external) that builds up the        dictionaries and patterns based on processing text corpora        (either labeled or unlabeled),    -   built-in mechanisms allowing system's users to correct the data        manually and collecting their activities (those can be used        directly or as a training dataset for machine learning        correction modules).

Common Typos (Rule-based) 304B utilizes trial and error method forreplacing common patterns that are often misspelled, e.g. “-ht” into“-th”, “-in” into “-ing”, “ae” into “ea.” The modified words arevalidated against the available dictionaries.

General Typos 304C is the main submodule combining various techniquesfor correcting typos. Its core comprises the measurement of worddistances that can be further parametrized according to the source ofdata and other aspects (e.g. cost of letter changes based on keyboardarrangement, sensibility to bad language, homophones). The parameterscan be adjusted manually, semi-automatically or automatically, based onprocessing of unlabeled datasets from a given data source.

Word Boundaries 304D resolves situations where the boundaries of wordsare incorrectly changed. The submodule takes into considerationintentional changes, including:

-   -   splitting (e.g. “you are s t u p i d.”),    -   joining (e.g. “youarestupid.”),    -   combination of splitting and joining (e.g. “y ouares t u p        id.”).

Additionally, the submodule resolves ambiguous usages of dash andhyphen, detecting its desired function and deciding if it should beremoved, preserved, changed from one to another (e.g. dash to hyphen) orchanged into other more suitable character (e.g. colon).

Enumeration/Listing 304E resolves any (complete and partial) usage ofenumeration and listing in input text. The enumeration/listing markupsare removed or changed in order to not interfere segmentation or anyother operations that take into consideration sentence boundaries. Thesubmodule tries to make the best decision based on the type and supposedpurpose of the enumeration/listing. For example, enumerations/listingsof words and phrases are often separated with commas, whereas sentencesare separated with full stops. Furthermore, the submodule normalizes theusage of commas and conjunctions.

“Stars” (Letter Replacement) 304F resolves situations where one or moreletters in a word is/are replaced with other non-letter characters (e.g.“*”, “#”, “.”). This kind of replacement within abusive words is oftenused to mislead the detection system. The submodule allows to correcteven the words with altered beginnings as well as with the combinationof different characters. It validates 1-to-1 replacement as well as anyother options such as 1-to-many, many-to-1, many-to-many,1-or-many-to-zero.

Leet Speak 304G resolves letter replacements related to so-called leetspeak, a system of modified spellings. Leet speak often replaces letterswith similarly looking characters or modifies words based on a system ofsuffixes and alternate meanings. The submodule replaces the replacementswith the correct letters, e.g. “1 h4t3 y0v” is changed into “I hateyou”, “1 h0p3 u d13” into “I hope you die.”

Part-of-speech Agreements 304H resolves syntactic errors based ondesirable part-of-speech patterns. The submodule makes use of the factthat certain combinations of parts-of-speech in certain order are highlyunlikely or even impossible. Examples: “you are real stupid” into “youare really stupid”, “their a bunch of idiots” into “there are a bunch ofidiots.”

Other Correction 304I covers every other correction that does not fallinto the above-mentioned modules.

As default, Correction 304 performs two passes in order to spot anycombination of the above-mentioned phenomena. Furthermore, thesubmodules can overlap and therefore it is possible to correct anyreasonable combination such as Leet Speak 304G and Word Boundaries304D—“@ $ $ h 0 1 3” is changed into “asshole.” It is also possible toset triggers executing additional passes based on various parameters,e.g. percentage of dictionary-verified words.

Transformation 306 comprises the process of revising the corrected textin order to make it more suitable for Detection Engine 114. The maingoal of Transformation 306 is to remove ambiguity of input text byperforming well-defined string manipulations. FIG. 7 is a block diagramproviding detailed division of Transformation 306 into a set ofsubmodules realizing specific tasks.

Technical/Structural 306A provides changes within general structure oftexts and sentences, such as punctuation and segmentation-relatedindicators. There are two major aspects of this submodule:

-   -   sentence merging/splitting—if consecutive sentences make more        sense as a single sentence, or if a single sentence makes more        sense as separated sentences, they are merged together or split        up (additional connector can be added if required), e.g. “This        dress. Is beautiful. You are not.” is changed into “This dress        is beautiful, but you are not.”    -   comma insertion/deletion—if a comma is incorrectly omitted (or a        sentence is more comprehensible with a comma), or a comma is        redundant, the comma is added to/removed from the sentence, e.g.        “Read my page idiot.” is changed into “Read my page, idiot.”

To Abusive 306B replaces expressions (words, phrases, etc.) that areabusive in a given context but were softened or deprived of coarselanguage in order to hide the bad intention (euphemisms, idioms, etc.)with a single semantically equivalent expression that means the same andis unambiguous (or strictly abusive), e.g. “up your bottom” is changedinto “up your ass”, “polishing the spear” into “masturbating.” All ofthe semantic transformations are convertible and preserve all theavailable information such as appropriate grammar forms such as verbforms, e.g. “i saw you shook your meat yesterday” is changed into “i sawyou masturbated yesterday.”

To Non-abusive 306C is a reciprocal of To Abusive 306B. It replacesexpressions that are not abusive but contain bad language with clearlynon-abusive expressions. It lowers the probability of incorrectinterpretation of such expressions. For example, “raping your inbox” ischanged into “spamming your inbox”, “being the shit” into “being thebest.” All of the semantic transformations are convertible and preserveall the available information.

To Be Deleted 306D removes expressions (words, phrases, etc.) that areirrelevant to the meaning in a given context, such as part ofinterjections (e.g. “wow”, “oh”), part of internet slang abbreviations(e.g. “lol”, “rofl”), part of onomatopoeias (e.g. “blah blah”, “hahaha”)and word “that” when it can be omitted (e.g. “i hope that you die” ischanged into “i hope you die”). Furthermore, the module can detectabusive words used as intensifiers and remove them as well, e.g. “youare fucking awesome” is changed into “you are awesome.” Alternatively,the intensifiers can be handled by the To Non-abusive 306C module.

Coreference Resolution 306E replaces referring expressions such aspronouns with the right entities that they refer to. It simplifies thedetection process as usually the simpler relations need to be detected.The submodule operates on any referring expressions, not only pronouns.It also allows to distinguish nominal and adjectival aspects of theentities. For example, “jews are greedy. i hate their religion.” ischanged into “jews are greedy. i hate jewish religion.”, “i saw your momyesterday. she is fat like a pig.” into “i saw your mom yesterday. yourmom is fat like a pig.”

Omission Fulfillment 306F fulfills (usually using “copy and paste”method) expressions (words, phrases, etc.) that can be omitted innatural human communication, but their absence can be problematic forthe system to understand the context, including (but not limited to):

-   -   cloning subject+part of predicate if needed, e.g. “you need to        crawl under a rock and die” is changed into “you need to crawl        under a rock and you need to die”,    -   cloning subject+predicate, e.g. “why do not you send me $1000.        if not, i will kill your dog.” is changed into “why do not you        send me $1000. if you do not, i will kill your dog.”,    -   cloning infinitive particle “to”, e.g. “i am going to hunt you        down and murder you violently” is changed into “i am going to        hunt you down and to murder you violently.”

Additionally, Omission Fulfillment 306F fulfills words that areincorrectly omitted, based on a validation of grammar correctness. Thealgorithm fulfill the lack with the most probable word based on a givencontext. This aspect is similar to Part-of-speech Agreements 304H fromCorrection 304 module. For example, “you an idiot” is changed into “youare an idiot”, “you should nice to your mom” into “you should be nice toyour mom.”

Entity Conversion 306G detects named entities and other strings that canbe considered to be abusive in a given context, including (but notlimited to):

-   -   person/brand/music band names,    -   song/movie/book/TV show title,    -   famous quotes,    -   song lyrics.

Detected entities and strings are replaced with equivalents that areneutral to Detection Engine 114, e.g. “john tucker must die” is replacedwith “movie title.”

Grammar Construction 306H covers the transformation of a well-definedpart of the grammar constructions that either exclude the otherwiseviolent utterance from being detected or changes the non-violentutterances in order to become detectable by Detection Engine 114. Thereare three major constructions handled by this submodule:

-   -   quoting—using explicit utterances (direct speech) of other        people, e.g. “he said: ‘you are an idiot,’ and left the room.”    -   reported speech—expressing other people's utterances without        quoting them, e.g. “he said that you are an idiot.”    -   calling x y—this is a collective name for a group of        constructions that indirectly assign an abuse to an object, e.g.        “i am sorry for calling you an idiot.”

The other part of constructions excluding the otherwise violentutterance from being detected is covered by Contradiction Checking whichis broadly described in Detection Engine section.

Every grammar construction can be placed in any of these two submodules.A decision where a given construction should be placed (GrammarConstruction 306H or Contradiction Checking) depends on soft guidelines:

-   -   any form of expressing other people's utterances should be        placed in Grammar Construction 306H,    -   expressions that require to check both sides of the potentially        detected statement (what is before and after) should be placed        in Grammar Construction 306H,    -   counter-factual expressions should be placed in Contradiction        Checking,    -   negation-related expressions should be placed in Contradiction        Checking,    -   specific constructions (applicable to only a part of detection        models) should be placed in Contradiction Checking.

The information about detected grammar construction is passed throughthe whole detection process so it is possible for the system to tellthat the utterance contains certain category of online violence (e.g.direct personal attack), but the violence occurred in a quote. GrammarConstruction 306H can replace detected content with anything neutral toDetection Engine 114.

Other Transformation 3061 covers every other transformation that doesnot fall into the above-mentioned modules.

FIGS. 8-9 are diagrams illustrating a process of Text Preparation 112module according to an embodiment. Referring to FIG. 8, the input toText Preparation 112 module is the text “Ima cum 2 ur h0u$3 . . . &amp;\u2018mak sure\u2019 dat u w0n+have kids !!!1!” According toNormalization 302, five submodules are used: Character Encoding 302A,Redundant Repetitions 302B, Tags/Markers/Code Snippets 302C,Contractions 302E, and Whitespaces 302F. After Normalization 302, theinput text for Correction 304 is: “i am going to cum 2 ur h0u$3 . . .and “mak sure” dat u w0n+have kids!” According to Correction 304, threesubmodules are used: Common Typos (Statistical) 304A, General Typos304C, and Leet Speak 304G. After Correction 304, the input text forTransformation 306 is: “i am going to come to your house . . . and “makesure” that you will not have kids!” According to Transformation 306,three submodules are used: Technical/Structural 306A, To Be Deleted306D, and Omission Fulfillment 306F. The output of Text Preparation 112module is: “i am going to come to your house and i am going to make sureyou will not have kids!” Referring to FIG. 9, the input to TextPreparation 112 module is: “I HaT GAYZ . . . <br>tHe SHuOLD BuRyED iN DaGRoUnD . . . ” According to Normalization 302, two submodules are used:Redundant Repetitions 302B, and Tags/Markers/Code Snippets 302C.According to Correction 304, four submodules are used: Common Typos(Statistical) 304A, Common Typos (Rule-based) 304B, General Typos 304C,and Leet Speak 304G. According to Transformation 306, two submodules areused: Coreference Resolution 306E, and Omission Fulfillment 306F. Theoutput of Text Preparation 112 module is: “i hate gays. gays should beburied in the ground.”

Detection Engine

Referring briefly to FIG. 3, the output of Text Preparation 112 isreceived by Detection Engine 114 module. FIG. 10 is a block diagram ofDetection Engine 114 module architecture according to an embodiment. Inan embodiment, Detection Engine 114 comprises three submodules: ViolenceTowards Second Person Detection (VTSPD) 402, Transformations To SecondPerson (TTSP) 404, and Other Violence Detection (OVD) 406. According toan embodiment, only Violence Towards Second Person Detection (VTSPD) 402is obligatory, the other two submodules are optional, but allow toexpand the detection process to online violence phenomena targetedagainst any other objects than interlocutor. Other embodiments allow toinclude more submodules, following the same architecture described inthis document.

Violence Towards Second Person Detection (VTSPD) 402 comprises the coreof Detection Engine 114 as potentially the most damaging online violencephenomena are targeted against an interlocutor. VTSPD utilizes a set ofcontextual models for detecting certain types of online violence. Acontextual model can be seen as a complete functioning detection modelfor detecting a single well-defined category of online violence.Although the contextual models are independent, they can overlap eachother (e.g. a given violent sentence can be detected by more than onecontextual model). Furthermore, contextual models can utilize the samesystem's assets such as dictionaries or rule libraries. According to anembodiment, contextual models related to certain type of online violenceare grouped together as a submodule of Detection Engine 114, definingthe target of online violence—an interlocutor in case of ViolenceTowards Second Person Detection (VTSPD) 402. Other embodiments maycomprise different functional structure.

FIG. 11 is a block diagram providing detailed division of ViolenceTowards Second Person Detection (VTSPD) 402 into a set of contextualmodels detecting certain types of online violence. As contextual modelsare independent from each other, their order is completely arbitrary.All of the following contextual models are presented as components ofVTSPD and therefore the terms “interlocutor” and “second person” areused to point out the target of online violence. Aside from obvioussecond person indicators (such as pronoun “you”), VTSPD also coversonline violence targeted against interlocutor's relatives and otherpersons/things important for the interlocutor. Therefore, whenever theterms “interlocutor” and “second person” are used, the broaderdefinition should be concerned. According to an embodiment, violencetowards other persons is detected using VTSPD after performingTransformations To Second Person (TTSP) 404. Other embodiments maycomprise different functional structure.

Contextual models can be further divided into contextual sub-modelswhich detect particular online violence phenomena within a given type ofonline violence. The following description of contextual models containsexemplary divisions into online violence phenomena. Such division can bea resemblance of the division into contextual sub-models. For thedemonstration purpose, in case of two sub-models—Blackmails 402C andSexual/Child Exploitation 402E, the division into contextual sub-modelswill be presented explicitly. All examples in the following section arepresented as they would be after Text Preparation 112. It means that theexemplary texts are lowercase and do not have contractions,abbreviations, misspellings, and other elements corrected and normalizedin Text Preparation 112.

Personal Attacks 402A detects any forms of abusive remarks on orrelating to an interlocutor, including (but not limited to):

-   -   direct abuses, e.g. “you are an idiot”,    -   indirect abuses, e.g. “stop acting like an idiot”,    -   abusive comparisons, e.g. “you smell like a skunk”,    -   abuses using vocative case, e.g. “hold on, idiot”,    -   part of cyberbullying related with personal remarks, e.g. “your        parents were paid to adopt you.”

Threats 402B detects any forms of threats, bad intentions and bad wishestowards an interlocutor, including (but not limited to):

-   -   direct threats, e.g. “i am going to kill you”,    -   indirect threats, e.g. “sooner or later we will get you”,    -   implied threats, e.g. “i know where you live”,    -   intentions and wishes for harming an interlocutor, e.g. “i wish        you have aid”,    -   regrets for non-harming an interlocutor, e.g. “too bad your        family is not dead.”

Blackmails 402C detects blackmails, coercions and part of extortions.FIG. 12 illustrates the division of Blackmails 402C into contextualsub-models. Four contextual sub-models are disclosed (there are more):

-   -   revealing information, e.g. “give me money or your parents will        find out you are gay”,    -   damaging physical integrity, e.g. “give me money or I will kill        you”,    -   depriving of or damaging property, e.g. “hire me or i will set        fire to your car”,    -   humiliation, e.g. “give me your pocket money or i will clean the        bathroom floor with your clothes.”

Blackmails 402C contextual sub-models are therefore classified by thetype of threat in case of not fulfilling the blackmailer's request.Referring to FIG. 12, Blackmails 402C contextual sub-models appear intwo versions which is mapped into the output categorization. Eachsub-model can be detected as punishable or non-punishable—punishableblackmail and extortion acts typically use threats of criminalprosecution in accordance with the American law. This classification canbe adjusted to any other law system if needed. It is worth noting thatdue to the threat-related classification, Blackmails 402C shares thesystem's assets (dictionaries, libraries) with Threats 402B.

Sexual Harassment 402D detects any form of sexual harassment towards aninterlocutor, including (but not limited to):

-   -   proposing sexual-oriented activities, e.g. “i want you to give        me a blowjob”,    -   sharing sexual information/experiences, e.g. “i masturbate        thinking of you”,    -   asking inappropriate sexual-oriented questions, e.g. “what is        the size of your boobs?”,    -   sexually-oriented threats, e.g. “i will rape you until you        bleed”,    -   making inappropriate comments, e.g. “your bulge is too big for        my mouth.”

Sexual/Child Exploitation 402E detects any behaviors/activities aimingat exploiting an interlocutor sexually, especially targeted againstchildren. FIG. 12 illustrates the division of Sexual/Child Exploitation402E into contextual sub-models. Five contextual sub-models aredisclosed (there are more):

-   -   child grooming, e.g. “there is nothing wrong with hugging an        older guy like me”,    -   nudity extortion, e.g. “can you show me your picture of what is        under your skirt?”,    -   persuading sexual activity, e.g. “let us meet, i will teach you        how to satisfy men”,    -   intimate question, e.g. “have you had your period yet?”,    -   isolation, e.g. “why would you need your immature schoolmates        when you have an adult friend like me?”

Sexual/Child Exploitation 402E contextual sub-models are thereforeclassified by the type of behavior/activity. It allows to design arelevant reaction for each classification, e.g. informing a child thatit is completely normal to talk to the parents about everything if anattempt to isolation is detected. It is worth noting that due to thesex-related phenomena, Sexual/Child Exploitation 402E shares thesystem's assets (dictionaries, libraries) with Sexual Harassment 402Dand Sex Trafficking 402H.

Persuading Suicide/Self-harm 402F detects coaxing, urging, persuadingand encouraging of an interlocutor to commit suicide or to harm oneself,including (but not limited to):

-   -   direct coaxing, e.g. “you should kill yourself”,    -   conditional coaxing, e.g. “if i were you, i would jump off the        bridge”,    -   lowering self-esteem, e.g. “your life is worthless”,    -   rationalizing, e.g. “binging and purging are fine, everyone does        that”,    -   presenting self-harm as something good, e.g. “you should try        slitting your wrists, it is such a relief.”

Persuading Violence 402G is analogous to Persuading Suicide/Self-harm402F and detects coaxing, urging, persuading and encouraging aninterlocutor to use violence (either verbal or physical) towards otherpeople or animals, including (but not limited to):

-   -   direct coaxing, e.g. “we should smash his face”,    -   conditional coaxing, e.g. “if i were you, i would kick his ass”,    -   lowering self-esteem, e.g. “i can not believe you allow him to        talk this shit about you”,    -   rationalizing, e.g. “that is fine, he even does not deserve to        live”,    -   presenting violence as something good, e.g. “nothing will make        you feel better than beating this piece of shit.”

Sex Trafficking 402H detects any forms of baiting both minors and adultsto be forcefully involved in commercial sex acts through a detailedanalysis of requests, threats and offers, including (but not limitedto):

-   -   suspicious job offers, e.g. “i am looking for a young, female        model for an international photoshoot in egypt”,    -   suspicious matrimonial offers, e.g. “a 24-year-old man looking        for an under-20-girl, willing to take you on a long vacation”,    -   cross-country dating, e.g. “i feel as if i have known you my        entire life, come visit me here in dubai, i promise we will have        a great time together”,    -   luring for a better life, e.g. “life here is so much better, we        have everything that you could dream of. just come, i will help        you with everything”,    -   international sponsorship, e.g. “let us just spend some time        together, no strings attached, and i will get you something        pretty in return.”

Other Second Person Violence 4021 stands for any new contextualsub-model that can be defined and build in order to detect anotherwell-defined online violence phenomenon. The system does not set up anylimitation in regard to this matter.

Transformations To Second Person (TTSP) 404 comprises the method forusing Violence Towards Second Person Detection (VTSPD) 402 in order toonline violence targeted against any other individual or group(referenced as “new violence object” in this description). In general,the method consists of:

1. Removing/hiding the second person references (e.g. pronouns—“you”,“your”, “yourself”) in order to prevent repeat detections or not misleadVRSPD in other way. An example approach is to replace the second personreferences with responding form of the word “someone”, e.g. “you” with“someone”, “your” with “someone's.”2. Replacing references to the new violence object with responding formof second person references (e.g. pronouns—“you”, “your”, “yourself”) inorder to use exactly the same contextual models to detect onlineviolence towards the new violence objects. This step requires topreserve and pass through the original text and the information whichnew violence object was transformed.

FIG. 13 is a block diagram providing detailed division ofTransformations To Second Person (TTSP) 404 into a set of well-definedtransformations of certain semantic/syntactic references to the newviolence objects. The output of Transformations To Second Person (TTSP)404 is the input to the next pass of Violence Towards Second PersonDetection (VTSPD) 402. There is no limit of subsequent passes. Everytransformation should be followed by a separate pass of Violence TowardsSecond Person Detection (VTSPD) 402 in order to not confuse thedetection process and not lose the ability to precisely say whichindividual or group is the target of online violence detection.

Personal Pronouns 404A is the basic transformation replacing thirdperson personal pronouns (“he”, “she”, “they”, “his”, “himself”, and soon) with responding second person personal pronouns (“you”, “your”, andso on). It is very important to note that this transformation is appliedafter Coreference Resolution 306E from Transformation 306 in TextPreparation 112 module described in details in the previous section. Theusage of Coreference Resolution 306E greatly reduces the chance ofreplacing unwanted pronouns as—at this point—part of the pronouns isalready replaced with responding entities. The term “unwanted pronouns”refers to the situation that should not be detected as violence, e.g.expressing opinions about unspecified group of people such as “somepeople even do not realize they are idiots.” Part of such situations areresolved by Coreference Resolution 306E, e.g. the above-mentionedsentence would be changed into “some people even do not realize somepeople are idiots.” Additionally, the transformation has a built-inmethod for verifying if a given third person personal pronoun should beignored as it refers to unspecified individual or group.

Named Entities 404B replaces named entities that can be objects ofonline violence with responding second person references. Thistransformation can be further divided into various functional groups.The first and the most obvious group consists of persons identified bynames (full, first names, last names, etc.). The transformation allowsto treat differently private persons and public figures such aspoliticians, celebrities, and so on. The user of the system (e.g. clientusing the API) is provided with the information against whom onlineviolence was used and can decide what to do with this information onone's own. The other groups consist of members of named groups or namedgroups themselves. This covers the combination named person/groups andmembership indicator defined by common nouns, e.g. “Trump supporters” or“fans of SF Giants.” This submodule has an access to the system'sassets, including dictionaries and statistical classifiers. According toan embodiment, there are built-in methods for performing Named EntityRecognition (NER) combining both statistical and symbolic processing.Other embodiments allow to use external NER tools, e.g. accessed throughan API.

Minority Terms 404C replaces terms related to various minorities withresponding second person references. This transformation can be furtherdivided into various functional groups, including (but not limited to):people of given sexual orientation, ethnicity, nationality, religion,supporters of given political party, people representing specific views,people practicing specific profession. In an embodiment, the minorityterms are detected and replaced using dictionary-based methods. Otherembodiments allow to use dedicated statistical classifiers in order toperform this task.

Other Group/Individual Terms 404D is very similar to Minority Terms404C, but covers the replacements of different terms, mainly related togroups that are not considered as minorities in the Western world,including (but not limited to): people representing certain nations orreligions (e.g. Americans, Christians). Other Group/Individual Terms404D and Minority Terms 404C use the same methods of detection andreplacement, according to an embodiment. Although, OtherGroup/Individual Terms 404D is further constrained by additionalcontext-based rules in order to avoid over-reacting and not violate thefreedom of speech in any way.

Other Transformation 404E stands for any new transformation that can bedefined in order to include another well-defined individual or groupinto the standard process of online violence detection. The system doesnot set up any limitation in regard to this matter.

Other Violence Detection (OVD) 406 resembles the same methods fordetecting online violence as Violence Towards Second Person Detection(VTSPD) 402, but functionally covers every other aspect of onlineviolence other than violence targeted against an interlocutor, or anyother individual and group, but only if the violence resembles violencetargeted against second person. Therefore, it is very important to notethat an overall violence detection regarding certain group of people(e.g. Jews) is usually split between two submodules:

1. Violence Towards Second Person Detection (VTSPD) 402 through theusage of Transformations To Second Person (TTSP) 404, which coversphenomena such as personal attacks or threats against certain group(e.g. Jews).

2. Other Violence Detection (OVD) 406, which covers other phenomena,such as the Holocaust denial and praising Hitler/Nazism in case ofdetecting violence towards Jews.

FIG. 14 is a block diagram providing detailed division of Other ViolenceDetection (OVD) 406 into a set of contextual models related to (anddetecting) certain types of online violence. Similarly to ViolenceTowards Second Person Detection (VTSPD) 402, Other Violence Detection(OVD) 406 utilizes a set of independent contextual models. According toan embodiment, the OVD's contextual models are built using the samemethodology and having an access to the system's assets such asdictionaries and libraries, exactly as the contextual models of VTSPD.

Sexting/Sex-related Content 406A detects if a given text contains anysex-related expressions. In comparison to sex-related contextual modelsfrom Violence Towards Second Person Detection (VTSPD) 402 (SexualHarassment 402D, Sexual/Child Exploitation 402E, Sex Trafficking 402H),Sexting/Sex-related Content 406A can be seen as the broadest sex-relatedmodel as the detection does not require the violence object, only thesex-related content. Although, VTSPD modules can still detect phenomenathat are not (and should not be) detected by Sexting/Sex-related Content406A. In this manner, all those models are complementary. The mainapplication of Sexting/Sex-related Content 406A is to protect childrenfrom the content that they should not be presented with. According to anembodiment, this contextual model almost exclusively utilizesstatistical classification. In this case, the governance of the symbolicpart is boiled down to launch the statistical classifiers and to verifythe symbolic exceptions that may exclude a text from detection due tosome very specific conditions (e.g. detecting school materials about sexeducation if they are allowed).

Antisemitism 406B detects hostility to, prejudice, or discriminationagainst Jews. As mentioned, it focuses solely on phenomena not coveredby the VTSPD's contextual models after specific transformation.Antisemitism 406B can be divided into contextual sub-models representingwell-defined phenomena, including (but not limited to):

-   -   stereotypes/opinions that are not abusive if related to ordinary        second person, but become abusive if related to Jews, e.g. “jews        support only other jews”,    -   praising or justifying Hitler/Nazism, e.g. “hitler would be a        perfect leader for our nation”,    -   antisemitic slogans and taglines, e.g. “warm up the gas        chambers”,    -   abusive antisemitic terms, e.g. “kike”,    -   Holocaust denials/lies, e.g. “there is no proof that holocaust        has ever happened.”

Self-harm Declaration 406C detects declarations/intentions of the personwho wrote the message to harm him- or herself, including committingsuicide. This contextual model does not have its supplement VTSPDcounterpart as it does not detect online violence targeted towards anyother individual or group. According to an embodiment, it is built usingthe same methodology as other contextual models. The model consists ofcontextual sub-models related to various form ofdeclarations/intentions, including (but not limited to):

-   -   explicit declarations, e.g. “i am going to kill myself”,    -   implicit intentions, e.g. “my life does not make any sense to me        anymore”,    -   self-convincing/justifying, e.g. “if i die, no one would even        care”,    -   dangerous considerations, e.g. “maybe the better choice would be        jumping off the bridge?”

X-ism/Y-phobia 406D detects phenomena such as sexism, racism, ageism,ableism, homophobia, and any other hostility to, prejudice, ordiscrimination against minorities. Similarly to Antisemitism 406B,X-ism/Y-phobia 406D covers all of such phenomena not covered by theVTSPD's contextual models after specific transformation, including (butnot limited to):

-   -   stereotypes/opinions that are not abusive if related to ordinary        second person, but become abusive if related to certain        minority, e.g. “all muslims are terrorists”,    -   spreading harmful conspiracy theories, e.g. “gays control the        mainstream media in my country”,    -   X-ist/Y-phobic slogans and taglines, e.g. “gay as aids”,    -   X-ist/Y-phobic terms, e.g. “faggot”,    -   X-ist/Y-phobic jokes, e.g. “q: why did the polack put ice in his        condom? a: to keep the swelling down.”

Other Violence 406E stands for any new contextual sub-model that can bedefined and build in order to detect another well-defined onlineviolence phenomenon. The system does not set up any limitation in regardto this matter.

Contextual Models

Contextual models are the foundation of the system and method. Acontextual model detects a single well-defined type of online violenceor a single well-defined online violence phenomenon. However, thisdivision is completely conventional. It is assumed that type of onlineviolence is a superordinate category detected by contextual models (e.g.Sexual/Child Exploitation 402E), whereas online violence phenomenon is asubordinate category detected by contextual sub-models (e.g. nudityextortion). Under this assumption, contextual sub-models detectingonline violence phenomena are grouped together into superordinatecontextual models detecting types of online violence. For reasons ofconvenience, in this section both contextual models and contextualsub-models are referred as contextual models because the above-mentioneddivision is only the result of introducing the auxiliary hierarchy.

The contextual model is an implementation of a dedicated algorithmdesigned to:

a) process an input text preprocessed with Text Preparation 112 modulein order to

b) detect a selected well-defined online violence phenomenon.

According to an embodiment, contextual models are built using thearchitecture and methodology disclosed in this document.

FIG. 15 is a block diagram of general architecture of online violencedetection modules according to an embodiment. Every contextual modelutilizes the presented architecture. Each contextual model takes in aText Preparation Output 308 as an input, and returns Detection Output570 as an output of the processing. According to an embodiment, TextPreparation Output 308 and Detection Output 570 are the samearray/object with the fields/variables (related to the result) fulfilledafter the detection process. In an embodiment, the output array/objectis created in the initial module and then it passed through each module,and each module adds new information (fulfills certain fields/variables)related to the process of the module. In an embodiment, the outputarray/object utilizes the following form (fields/variables withresponding type in parenthesis):

-   -   “text_orig” (string),    -   “text” (string),    -   “segmentate” (array of strings),    -   “parse” (array of arrays),    -   “action” (array or arrays),    -   “result” (array of arrays).

The fields “text_orig” and “text” are fulfilled in Text Preparation 112and they refer to the original input text and the text modified withinthe module, respectively. The fields “segmentate” and “parse” arerelated to syntactic parsing, and are fulfilled in various stages ofText Preparation 112 (“parse” can be fulfilled in Detection Engine 114).The field “action” is an additional field that stores the informationregarding particular text manipulations performed in Text Preparation112 module. Exemplary entry in “action” field: “[”133t speak“]=>array(1){[0]=>string(14) “idi0t=>idiot” }” refers to the correction of word“idi0t” to “idiot” using Leet Speak 304G from Correction 304 module. Thefield “result” remains empty in Text Preparation 112 and is fulfilled inDetection Engine 114 by the process of contextual models. Exemplaryentry in “result” filed: “array(l) {[0]=>array(3) {[0]=>string(4) “high”[1]=>string(10) “#peratt001” [2]=>string(9) #id000001”}}” refers to theprocessing of the sentence “You are an idi0t.” According to anembodiment the above-mentioned example of field “result” contains:

-   -   “[0]=>string(4) “high”” refers to the level of violence (“high”,        “medium”, “low”),    -   “[1]=>string(10) “#peratt001”” refers to the internal violence        category, here “#peratt001” is a token referring to direct        abuses within Personal Attacks 402A,    -   “[2]=>string(9) “#id000001”” refers to the specific rule        responsible for the detection, here “#id000001” is a token        referring to the rule finding abuse assignment to second person        using linking verbs.

It is important to note that according to an embodiment, each detectioncan be easily tracked down to the single rule responsible for thedetection, using the tokens referring to particular contextual modelsand to particular rule within this model. Furthermore, every actionperformed by Text Preparation 112 module is stored and available for theanalysis or development purpose. Therefore, contrary to prior artblackbox approaches based on machine/deep learning, the system andmethod propose a largely explainable and trackable decision-makingprocess.

Referring to FIG. 15, the main block is Symbolic Processing 510,responsible for:

-   -   executing symbolic algorithms, methods and components, including        detection rules, and    -   governing the whole detection process, including requesting        other system's components and gathering data from them,    -   fulfilling the result-related fields/variables in Detection        Output 570.

Symbolic Processing 510 is described in details later in this document.

Syntactic Parser 520 is responsible for providing information regardingsyntactic analysis to the system. The decent set of informationdelivered by Syntactic Parser 520 consists of the division into words,phrases and clauses and the syntactic relations between them, includingdependencies (e.g. phrase A is subject to phrase B) and inclusions (e.g.phrase A contains words X, Y and Z). According to an embodiment,Language Decoder (as described in U.S. Pat. No. 9,152,623 issued Oct. 6,2015 and owned by Applicants) is used as a syntactic parser providingoptimal amount of information as well as high accuracy for processinguser-generated content. Other embodiments allow to use differentsyntactic parsers, including various implementations of dependency orphrase structure grammar parsing. Although, the complement of lackinginformation may be required in any form (e.g. a syntactic parser'swrapper or a set of component rules). The output of Syntactic Parser 520can be utilized directly with the use of a general purpose programminglanguage or using a dedicated query language. The latter is recommendedas it greatly eases the process of building rules using parser's output.According to an embodiment, both the component rules and the detectionrules are built using a dedicated query language designed to work withLanguage Decoder. Other embodiments allow to use other methods forbuilding rules. Using other syntactic parser may require to use adifferent query language or to adjust the one designed to work withLanguage Decoder to work with another parser. If a syntactic parserprovides additional information such as grammatical tense (e.g. presentsimple), mood (e.g. imperative) or word's base forms (e.g. “hate” as abase form of “hating”), it can be a great aid for the process ofbuilding rules. According to an embodiment, Language Decoder providessuch information. Other embodiments may require to build dedicated rulesfor establishing grammatical tense, mood or word's base form.

FIG. 17 is a diagram illustrating syntactic parser (Language Decoder)output according to an embodiment. 502A shows a visualization ofsimplified output of Language Decoder for the input text “Send me morenudes or I'll publish the ones I already have.” One can see the divisioninto words, phrases and clauses as well as the relations betweenphrases. The relation types are presented with labels for clauses andblock colors for phrase. Such visualization can be useful for gettingthe general impressions in regard to the syntactic decomposition. 502Bshows a part of a dump of the actual Language Decoder output accordingto an embodiment. The disclosed part provides the complete informationabout the first word (token) for the same input text as in 502A. One cansee that aside from the information regarding divisions and relations,there is also an information regarding:

-   -   original writing (“word_orig_lex”),    -   base form of words (“phrase_core_base”),    -   grammatical tense (“clause_tense”),    -   grammatical mood (“clause_constructions),    -   Named Entity Recognition (“ner_id”, “ner_type”, “ner_content”),    -   additional semantic information about phrases (“phrase_aspect”        and “phrase_category”).

Statistical Classifiers 530 consist of a set of dedicates statisticalclassifiers utilized mainly in the detection process, but also withinText Preparation 112 (e.g. Named Entity Recognition). The statisticalclassifiers mostly comprise machine/deep learning models trained onlabeled data in order to preform the assumed classification. However,any other approach is accepted as long as it realizes the assumedfunctionality. There can be more than one classifier realizing exactlythe same goal. An engineer can select which one should be used incertain contextual models, or even can use all of them and setconditions for selecting the best one under given circumstances. Theseinstructions are coded in the symbolic rules. Statistical Classifiers530 are organized in a form of library available to Symbolic Processing510. The symbolic rule can assign a part of the input text, send it tothe selected classifier as a request, and gets back the classifier'soutput as a response (e.g. a category and confidence score). Therefore,Statistical Classifiers 530 do not need to be physically on the samedevice as the rest of the system. According to an embodiment StatisticalClassifiers 530 are installed on the same device in order to minimizethe communication overhead. According to other embodiment, SymbolicProcessing 510 communicates with Statistical Classifiers 530 through theInternet via Application Programming Interface (API). The same ruleapplies to Syntactic Parser 520, and potentially all the othercomponents disclosed on FIG. 15. It is also possible to scatter theStatistical Classifiers 530, having part of them internally on the samedevice, and another part on an external server. Furthermore, it ispossible to use completely external services providing the statisticalclassifiers.

According to an embodiment, each of Statistical Classifiers 530 takes inan input text (can be only a part of the whole text from TextPreparation Output 308) and returns a set of detected categories (can bea single category for a binary classification) along with respondingconfidence score (from 0 to 1). Threshold for the confidence scores isset within Symbolic Processing 510. Statistical Classifiers 530 areorganized in a form of library and can be described with the training-and performance-related data, e.g. what was the training dataset (size,source), what architecture/learning algorithm was used, what is theperformance (average processing time, precision, recall, F1-score). Thisdata allows an engineer to select the most suitable classifier for thegiven purpose. In order to give the sense of the functionality range ofStatistical Classifiers 530, one can consider the following examples:

1. A text-level binary classification for detecting sexting/sex-relatedcontent: As mentioned in the description of Sexting/Sex-related Content406A, the statistical classification plays the dominant role in thiscontextual model. The classifier takes in the whole input text andreturns a confidence score (range from 0 to 1), describing the chancesfor the input text to contain any sex-related content (1 stands for veryhigh confidence).2. A clause-level binary classification for detecting negativeconsequences of a blackmail: In this case, the detection rules fromSymbolic Processing 510 assign a clause or a combination of clauses(obtained from Syntactic Parser 520 output) that potentially describesthe consequences of not fulfilling a request. This potential consequenceis then verified whether it is negative or not.3. A phrase-level multi-category classification for detecting abusivephrases/expressions: The detection rules from Symbolic Processing 510assign a phrase or a combination of phrases and/or clauses (obtainedfrom Syntactic Parser 520 output) that is assigned to an interlocutor(e.g. using linking verbs). The classifier's goal is to classify whetherthe phrase(s) falls into one or more of the predefined categoriesrelated to interlocutor's attribute (appearance, intelligence,knowledge, experience, and so on).

Rule Libraries 540 contain the rules utilized and executed in SymbolicProcessing 510. The rules can utilize Syntactic Parser 520, but do notneed to. According to an embodiment, there are two major types of therules:

-   -   detection rules are the rules detecting the smallest subpart of        the given phenomenon, such as detecting one of many ways how one        person can threat another person, e.g. using a second person        reference as a subject, and expressing the future using any        future tense or “be going to” types of constructions; therefore,        executing a detection rule can tell if the input text contains        this very specific subpart of online violence or not,    -   component rules consist of all the other rules realizing        specific sub-tasks (even non-obvious) related to the detection        of the given phenomenon, but not the phenomenon itself;        component rules are subparts of detection rules and can be seen        as building blocks; in other words, they can be used to compose        the detection rule from smaller parts realizing certain        well-defined tasks; some examples of component rules: detecting        requests, greetings, recommendations, wishes, regrets, second        person indications, intents, future expressions, and so on;        other component rules can be seen as wrappers responsible for        communication with Statistical Classifiers 530, including        assigning specific parts of the input texts, sending them to        selected classifier, and getting back the results.

Dictionaries 550 are lexical databases storing words, phrases, and otherexpressions used in the detection process. Functionally, they are usedto supplement the rules form Rule Libraries 540, as the rules focus ongrammar constructions and relations between entities, whereasDictionaries 550 provide the vocabulary. Therefore, the detection rulesare usually built from the component rules supplemented by theexpressions from Dictionaries 550 and the statistical classificationsfrom Statistical Classifiers 530. Dictionaries 550 are not limited tosingle words. It is possible to store the whole expressions. Although,for reasons of convenience of further development, in some cases it isbetter to split the expression into predicates and objects, and let therules combine them together. Some examples of dictionaries showing therange of potential applications:

-   -   verbs (including phrasal verbs) related to revealing an        information,    -   verbs and verb phrases related to harming a person        physically/verbally with extended categorization, including the        level of harmfulness (to hurt, to damage, to kill),    -   nouns and noun phrases related to male/female genitals with        extended categorization, including the abusiveness of the        expression,    -   sexual activities: from single words to complex expressions,        stored in a minimal form and allowed to be completed and/or        fulfilled with other words/expressions by the component rules,        e.g. an entry “verb: suck+noun: male_genitals” can be completed        with nouns/noun phrases related to male genitals and fulfilled        with descriptors allowed to describe the given nouns/noun        phrases. According to an embodiment, Dictionaries 550 are stored        in internal databases. Although, they can be built using        external sources. Other embodiments can utilize external        databases such as WordNet, OpenCyc or DBpedia. According to an        embodiment, dictionary entries are stored only in a base form.        If required, every kind of inflection, such as conjugation or        declination, is applied in Symbolic Processing 510 according to        the rule requirements. Other embodiments can utilize other        methods of storing entries in Dictionaries 550.

Unsupervised Learning Algorithms (ULA) 560 is a collective name foradditional models used for extending dictionaries and rules without anyneed for labeling data. Referring to FIG. 15, there are two instances ofUnsupervised Learning Algorithms (ULA) 560. The left-hand instanceaffects Dictionaries 550 and will be referred as Dictionary ULA. Theright-hand instance affects Symbolic Processing 510 and will be referredas Rule ULA. Both Dictionary ULA and Rule ULA utilize Unlabeled Data,which refers to any normal data set or data stream without humanlabeling. Although, in some cases pre-trained models can be used (e.g. apre-trained vector representation of words) and there is no need forusing any additional data. In these cases, the process of learning fromunlabeled data was just performed before, possibly by someone else. Eachof Unsupervised Learning Algorithms (ULA) 560 can be applied insemi-supervised way in order to decrease the number of rules ordictionary entries causing the drop in precision or recall. For example,ULA generated 18 new entries to a given dictionary: 11 correct, 6neutral (not affecting precision/recall) and 1 incorrect (causing asmall drop in precision). These entries can be just added to thedictionary (unsupervised process) or can be verified by a trained personin order to exclude the incorrect entry (semi-supervised process).

Dictionary ULA extends the existing dictionaries with new entries. Thealready existing dictionary entries are used as primers/seeds in thisprocess. It means that these entries comprise the starting set ofwords/phrases and the algorithms try to find different words/phrasesthat can play a similar role within similar context. The main methodconsists of using word embedding (vector representations of words). Ifneeded, the approach can be supplemented with the usage of lexicaldatabases and thesauruses (e.g. WordNet). In that case, there is atwo-step process of:

a) extending the list of primers/seeds with synonyms from selectedlexical database,

b) extending the list of primers/seeds/their synonyms with thealgorithms utilizing vector representations.

Alternative methods can be applied as long as they increaseprecision/recall and the consequential drop in the second parameter istolerable. The tolerance threshold can be set in a form of exchange par,e.g. 2 new true positives for 1 new false positive can be set as aminimal tolerance threshold, and verified on testing data set. Anexample of alternative method of Dictionary ULA is to use predefinedgrammar patterns (e.g. predefined list of subjects+linkingverb+arbitrary complement) utilizing the output of Syntactic Parser 520in order to create a list of extracted complements based on theprocessing of unlabeled data. The list of extracted complements can betherefore used to extend the pool of abusive terms or to improveCorrection 304 by finding similarly written but still misspelledentries.

Rule ULA extends mainly the detection rules, and the component rules tosome extent. It resembles the same idea—to use the existing rules asprimers/seeds and to generate new rules that realize the samefunctionality. It is important to note that the new rules have torepresent the same method of expression as primers/seeds. It means thatthey need to be executable using the same method/engine. Instead of wordembedding, a vector representation of syntactic trees delivered bySyntactic Parser 520 is used. A set of corresponding syntactic trees isgenerated for each of primer/seed rule. Then, the other trees arecompared to the ones delivered from existing trees, using vectorrepresentation built on large corpora of unlabeled data. Once thesimilar trees are selected, the algorithms try to compose a new rulebased on available elements, including component rules and dictionaries.Similarly to Dictionary ULA, alternative methods for generating newrules can be applied as long as they increase precision/recall and theconsequential drop in the second parameter is tolerable.

FIG. 16 is a block diagram providing detailed division of SymbolicProcessing 510 into a set of subsequent processing steps according to anembodiment. Each of the following steps has the full access to all ofthe components (Syntactic Parser 520, Statistical Classifiers 530, andso on). However, certain steps are strongly related to certaincomponents which will be emphasized in the following descriptions.

Preliminary Filtering 501 is the process of finding the input texts thathave no chance of being detected by given contextual model. These textsare excluded from the detection process of given contextual model, whichdrastically reduce the average processing time, especially in case ofrare phenomena detection. The main assumption is that there is a set offeatures that allows to decide with a very high confidence if a giventext has any chance for being detected or not. As the input text isalready corrected, in most cases Preliminary Filtering 501 relies on averification of a list of essential keywords, or a set of keywords inproper combination and order, e.g. one of filters may require 3 words inany form in a given combination where it does not matter what is betweenthese words. Therefore, Preliminary Filtering 501 utilizes any fast andreliable method for performing such search, e.g. regular expressions.According to an embodiment, every contextual model has a built-infunction that builds up the preliminary regular expression filtersautomatically, based on Rule Libraries 540 and Dictionaries 550 used inthe detection process. The automatically generated filters can be thenmodified by human engineers if needed. Some of online violence phenomenaare very rare. If there is a 1% of a phenomenon in normal datasets,cutting off 95% of texts that do not have any chance for being detectedis a huge profit.

Syntactic Parsing 502 is the process of using Syntactic Parser 520 inorder to provide the detection rules with syntactic analysis. Only theinput texts that passed through Preliminary Filtering 501 can be parsed.Furthermore, not every rule requires Syntactic Parsing 502. The inputtext is segmented at this point so each sentence can be parsedseparately. Additionally, Preliminary Filtering 501 can filter onlysentences, not the whole texts. It can be applied for the contextualmodels (or certain rules) that require only the sentence-basedinformation to operate. It is also possible to filter the sentence andthe whole text in order to parse the sentence and use the whole textdifferently, without parsing. According to an embodiment, the output ofSyntactic Parsing 502 is stored in the system's output array/object.Therefore, Syntactic Parsing 502 is performed at most once for eachsentence of the input text. Once the sentence is parsed in onecontextual model, the information is stored in the array/object andpassed through to the rest of models. Every contextual model verifies ifit has all of the required information and run Syntactic Parsing 502only if it is necessary.

Detection Rules Composition 503 is the process of combining together thecomponent rules and the entries from the dictionaries in order to formthe executable detection rules. The detection rules are build and storedin the most convenient way for their further development andmaintenance. Therefore, at this point their clarity and the ability toquickly adjust or modify them is more important than the processingspeed. The detection rules are coded using variables representingbuilding blocks, e.g. request+sexual activity. Detection RulesComposition 503 is the process of collecting the required data from allof the system's components and to substitute the conceptual elements ofthe detection rules with their executable counterparts. It covers theprocess of generating the required forms of words, phrases andexpressions stored in Dictionaries 550 in their base form. This processutilizes the built-in dedicated algorithms. For purpose of optimization,the executable detection rules are stored in Rule Libraries 540. If anyelement of the detection rule is changed, the new executable rule isgenerated. In contrast to development version of the system, theproduction version of the system requires only the executable detectionrules.

Detection Rules Execution 504 is the process of executing the executabledetection rules, delivered by Detection Rules Composition 503 or storedin Rule Libraries 540 if they were already processed.

This step largely overlaps with the two following steps: DedicatedClassifiers 505 and Contradiction Checking 506. Functionally, DetectionRules Execution 504 verifies if the input text matches the executabledetection rule. In that case, Detection Rules Execution 504 performs twoadditional steps:

1. Dedicated Classifiers 505: If the detection rule requires the usageof Statistical Classifiers 530, the selected part of the input texts isassigned as the classifier's input by Detection Rules Execution 504using the guidelines from the detection rule. Detection Rules Execution504 sends the request, gets the response and takes it into considerationexecuting the detection rules. At this point, Detection Rules Execution504 has the complete set of information required to detect onlineviolence.2. Contradiction Checking 506: If the input text is detected ascontaining online violence, the final step is performed. Detection RulesExecution 504 assigns the selected part of the input texts as the inputof Contradiction Checking 506. Usually, the input of ContradictionChecking 506 consists of the detected sentence and a couple ofsurrounding sentences. Although, sometimes the whole input text is used.Contradiction Checking 506 utilizes a special set of rules that verifiesthe broader context in order to find the indicators that the alreadydetected text should not be treated as online violence. These indicatorscan be any form of:

-   -   negation-related expression, “i don't think you are an idiot”,    -   counter-factual expressions, e.g. “at first, i thought you are        an idiot, but i was totally wrong”,    -   quotation or indirect speech, e.g. “i can not believe he said        you are an idiot”,    -   other expressions that do not fall into the above-mentioned        categories.

Functionally, the rules of Contradiction Checking 506 can be built andstored exactly as the detection rules. They use the same system'sassets: the component rules from Rule Libraries 540 and the entries fromDictionaries 550. Therefore, they can be substituted by their executablecounterparts in Detection Rules Composition 503 and stored in RuleLibraries 540. According to an embodiment, they are separated from thedetection rules due to the fact that the contradiction rules areindependent from the detection rules and the same set of contradictionrules can be applied to different contextual models. Therefore, thedetection rules and the contradiction rules can be developedindependently by different teams of engineers.

FIGS. 18-19 are diagrams illustrating a process of Symbolic Processing510 according to an embodiment. The two first steps—PreliminaryFiltering 501 and Syntactic Parsing 502—are skipped since they are notrequired to explain the process. One should assume that the input textsfrom both diagrams passed through Preliminary Filtering 501 and weresuccessfully parsed using Syntactic Parsing 502. Referring to FIG. 18,the input to Symbolic Processing 510 representing a contextual model ofPersonal Attacks 402A is the text “i can not believe he said you are anidiot.” Detection Rules Composition 503 is presented as a process ofcombining three elements in order to detect a personal attack (detectedelements are underlined):

-   -   second_person_indication from Rule Libraries 540 is a component        rule that allows to find second person references in text;        functionally, it detects if a given phrase contains a second        person reference; in the example “i can not believe he said you        are an idiot”, a reference is detected as subject “you”,    -   assignment_relation from Rule Libraries 540 is another component        rule that detects if there is an assignment relation between two        other elements; assignment relation is a relation that assigns a        feature or a set of features expressed within one element to the        entity expressed in the second element; the most frequent case        of assignment relation is using a linking verb that assigns the        features from complement to the entity from subject; in the        example “i can not believe he said you are an idiot”, an        assignment relation is detected as predicate “are”,    -   abusive_expression from Dictionaries 550 is a set of expressions        (words, phrases) that are considered to be offensive/abusive        while used against other person; the expressions can be        categorized with type of abuse or level of harmfulness;        Detection Rules Composition 503 takes the full set from        Dictionaries 550 and generates all required forms of the        expressions using the dedicated built-in functions; the enriched        set of expressions is then used to verify if a given phrase        contains an abusive expression; in the example “i can not        believe he said you are an idiot”, an abusive expression is        detected as complement “an idiot.”

The executable rule composed in Detection Rules Composition 503 step isthen executed in Detection Rules Execution 504 step. The personal attackis detected. Dedicated Classifiers 505 step is skipped since thisparticular contextual model does not require this step to operate. As afinal step, Contradiction Checking 506 is performed. According to theexample, a contradiction was detected (underlined) in a form of indirectspeech and therefore the input text is eventually not classified asonline violence, although it was detected with the executable rule.Symbolic Processing 510 returns “no violence” as a result of detectionprocess.

Referring to FIG. 19, the input to Symbolic Processing 510 representinga contextual model of Blackmails 402C is the text “send me more nudes ori will publish the ones i already have.” Detection Rules Composition 503is presented as a process of combining three elements in order to detecta personal attack (detected elements are underlined):

-   -   request from Rule Libraries 540 is a component rule that allows        to find any grammar construction that can be used to express        request; it can be simply an imperative or any of the following        constructions: “you should/have to/had better . . . ”, “why do        not you . . . ”, “if you do not . . . ”, and so on; the above        list serves the illustrative purpose, it is neither exhaustive        nor complete; in the example “send me more nudes or i will        publish the ones i already have”, a request is detected as        predicate (imperative) “send me more nudes”,    -   alternative from Rule Libraries 540 is another component rule        that verifies if two other elements are connected with each        other using a relation of alternative of not fulfilling a        condition; therefore, one of the two connected elements        comprises a condition (request), whereas the second comprises an        implication (consequence); the alternative does not need to by        explicitly present in text (e.g. “i will kill you if you do not        give me money”) to be detected with this component rule; in the        example “send me more nudes or i will publish the ones i already        have”, an alternative is detected as connector “or”,    -   get_consequence from Rule Libraries 540 is an example of        component rule that assigns part of an input text in order to        perform statistical classification using Dedicated Classifiers        505; this particular rule assigns the implication (consequence)        part of the alternative for the further processing (marked with        angle brackets in FIG. 19); in the example “send me more nudes        or i will publish the ones i already have”, a consequence is        assigned as “i will publish the ones i already have.”

During Detection Rules Execution 504 step, Dedicated Classifiers 505sends the assigned part of text to Statistical Classifiers 530, and getsback the response; here, the assigned text is classified as“reveal_information” with the confidence score of 0.99. Therefore, theblackmail is detected. Since Contradiction Checking 506 does not findany contradictions, Symbolic Processing 510 returns “blackmail,revealing information” as a result of detection process.

Customization

Due to the strict symbolic governance over the decision-making processand the clear modular architecture, the system and method demonstrateshigh customization capabilities. According to an embodiment, eachdetection rule, even if utilizes Statistical Classifiers 530, can beidentified by its unique ID number. Similarly, as mentioned in theContextual Models section, every modification introduced by TextPreparation 112 module is also stored and passed through the wholeonline violence detection process. Therefore, an engineer developing thesystem can immediately determine why the system made certain decision.This feature greatly reduces time and effort required for developmentand maintenance of the system.

According to an embodiment, a system developer is provided with adevelopment environment that allows to collect, visualize and analyzestatistics of processed data in regard to data sources and detectionrule IDs along with modifications introduced by Text Preparation 112module.

Furthermore, the development environment is equipped with the tools tolabel data and the ability to use labeled datasets in order to controlthe system's accuracy, but also to see which rules are responsible forcorrect and incorrect detections. This ability comprises a great aid inplaning the future development directions and adjusting the system toparticular data sources. Additionally, the development environment isintegrated with Syntactic Parser 520 which helps to track if an error isrelated to wrong parsing or not. In an embodiment, a system developeradjusts the system to given data source manually or semi-automaticallyusing the data stored in the development environment. In otherembodiments, the process of adjusting the system is fully automatizedutilizing labeled datasets and supervised machine learning methods.

The process of adjusting the system to particular data sources orguidelines is called “onboarding process.” The information related todata sources and guidelines can be obtained automatically as mentionedabove, or manually by referring to client's guidelines (if available) orinterviewing/surveying client's employees. The goal of theinterviews/surveys is to determine which online violence phenomena areallowed and which should be blocked within given service/community. Theinterviews/surveys can be performed by human (e.g. a system developer)or using predefined forms/applications (e.g. filling up a web-basedonline survey). The decisions regarding online violence phenomena areapplied by switching on and off the detection rules related to thesephenomena. This process can be performed manually, semi-automatically orautomatically. The most common types of settings regarding theonboarding process can be predefined and delivered through specific APIrequests. According to an embodiment, a control over what an API returnsin terms of the whole hierarchical output structure is provided withinthe API, using the API request features. Other embodiments require thedeployment of a dedicated API after adjusting the system to certain datasources or guidelines. Additionally, Unsupervised Learning Algorithms(ULA) 560 described in the Contextual Models section can be used as apart of onboarding process in terms of adjusting the system to certaindata sources.

The various functions or processes disclosed herein may be described asdata and/or instructions embodied in various computer-readable media, interms of their behavioral, register transfer, logic component,transistor, layout geometries, and/or other characteristics.Computer-readable media in which such formatted data and/or instructionsmay be embodied include, but are not limited to, non-volatile storagemedia in various forms (e.g., optical, magnetic or semiconductor storagemedia) and carrier waves that may be used to transfer such formatteddata and/or instructions through wireless, optical, or wired signalingmedia or any combination thereof.

Examples of transfers of such formatted data and/or instructions bycarrier waves include, but are not limited to, transfers (uploads,downloads, e-mail, etc.) over the internet and/or other computernetworks via one or more data transfer protocols (e.g., HTTP, FTP, SMTP,etc.). When received within a computer system via one or morecomputer-readable media, such data and/or instruction-based expressionsof components and/or processes under the system described may beprocessed by a processing entity (e.g., one or more processors) withinthe computer system in conjunction with execution of one or more othercomputer programs.

Aspects of the systems and methods described herein may be implementedas functionality programmed into any of a variety of circuitry,including programmable logic devices (PLDs), such as field programmablegate arrays (FPGAs), programmable array logic (PAL) devices,electrically programmable logic and memory devices and standardcell-based devices, as well as application specific integrated circuits(ASICs). Some other possibilities for implementing aspects of the systeminclude: microcontrollers with memory (such as electronically erasableprogrammable read only memory (EEPROM)), embedded microprocessors,firmware, software, etc. Furthermore, aspects of the system may beembodied in microprocessors having software-based circuit emulation,discrete logic (sequential and combinatorial), custom devices, fuzzy(neural) logic, quantum devices, and hybrids of any of the above devicetypes. Of course the underlying device technologies may be provided in avariety of component types, e.g., metal-oxide semiconductor field-effecttransistor (MOSFET) technologies like complementary metal-oxidesemiconductor (CMOS), bipolar technologies like emitter-coupled logic(ECL), polymer technologies (e.g., silicon-conjugated polymer andmetal-conjugated polymer-metal structures), mixed analog and digital,etc.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense as opposed to anexclusive or exhaustive sense; that is to say, in a sense of “including,but not limited to.” Words using the singular or plural number alsoinclude the plural or singular number respectively. Additionally, thewords “herein,” “hereunder,” “above,” “below,” and words of similarimport refer to this application as a whole and not to any particularportions of this application. When the word “or” is used in reference toa list of two or more items, that word covers all of the followinginterpretations of the word: any of the items in the list, all of theitems in the list and any combination of the items in the list.

The above description of illustrated embodiments of the systems andmethods is not intended to be exhaustive or to limit the systems andmethods to the precise forms disclosed. While specific embodiments of,and examples for, the systems components and methods are describedherein for illustrative purposes, various equivalent modifications arepossible within the scope of the systems, components and methods, asthose skilled in the relevant art will recognize. The teachings of thesystems and methods provided herein can be applied to other processingsystems and methods, not only for the systems and methods describedabove.

The elements and acts of the various embodiments described above can becombined to provide further embodiments. These and other changes can bemade to the systems and methods in light of the above detaileddescription.

What is claimed is:
 1. A system for detecting one or more predeterminedonline behaviors, the system comprising: at least one processorreceiving online data from any sources on the internet and processingthe online data to determine whether the online data contains onlineviolence, wherein processing the data comprises, the at least oneprocessor receiving plain text input; the at least one processorperforming a plurality of functional modules, the plurality offunctional modules comprising one or more of a text preparation module,a normalization module, and a correction and transformation module,wherein, normalization comprises the process of transforming the plaintext input into a single predetermined canonical form to generatenormalized text; correction comprises the process of revising thenormalized text in order to correct misspellings and any other types ofintentional and non-intentional errors to generate corrected text; andtransformation comprises the process of revising the corrected text inorder to replace, remove or add specific text fragments to generatetransformed text that is optimally phrased for detecting one or morepredetermined online behaviors; and the at least one processorperforming online violence detection on the normalized text, thecorrected text, and transformed text, including, at least one moduledetecting online violence, wherein the at least one module is comprisedof a set of contextual models wherein, each contextual model comprises adetection model for detecting a single well-defined type of onlineviolence; and each contextual model classifies text as containing or notcontaining a given type of online violence and assigns a set ofprocessing-related information to the text.
 2. The system of claim 1,wherein the processing-related information comprises: categories andsub-categories of detected violence; information allowing identificationof a rule responsible for the detection; and text preparation data. 3.The system of claim 1, wherein each contextual model comprises one ormore of symbolic processing, syntactic parsing, statistical classifiersand unsupervised learning techniques in detecting the one or morepredetermined online behaviors.
 4. The system of claim 3, whereinsymbolic processing of each contextual model: executes symbolicalgorithms, methods, and components, including detection rules, indetecting the one or more predetermined online behaviors; and governsusage of rules, dictionaries, outputs of a syntactic parser and thestatistical classifiers, including requesting supportive systemcomponents and gathering data from the supportive system components. 5.The system of claim 3 wherein symbolic processing of each contextualmodel comprises: preliminary filtering; composition and execution ofdetection rules; syntactic parsing; dedicated classifiers; andcontradiction checking.
 6. The system of claim 3, wherein symbolicprocessing of each contextual model comprises using one or more rulesstored in one or more rule libraries, wherein: each rule performs abinary classification in order to verify if a text input characterstring contains a sought language or online violence phenomenon; andeach rule has access to, and can make use of a syntactic parser,dictionaries and statistical classifiers.
 7. The system of claim 6,wherein each rule represents one of four functional categoriescomprising: detection rules identifying a separated and well-definedsubpart of a given online violence phenomenon, defined as a singlemanner in which a person can use natural language to express the givenphenomenon; component rules related to a specific sub-task for detectionof the given online violence phenomenon but not identifying the givenonline violence phenomenon itself and comprises a building block fromwhich the detection rules can be composed; contradiction checking rulesanalyzing a broad context of the input text and identifies thelinguistic premises indicating that the text should not be detected asonline violence regardless of the detection rules; wrapper rulesresponsible for communication with statistical classifiers, includingassigning specific parts of an input text character string, sending themto selected classifier, and getting back the results.
 8. The system ofclaim 3, wherein each statistical classifier complements the contextualmodels with text classification that verifies whether an input characterstring belongs to one or more predefined categories by taking in thestring and returning a set of detected categories along with acorresponding confidence score.
 9. The system of claim 3, whereinunsupervised learning techniques utilize unlabeled data ormachine-labeled data to improve the contextual models by increasing thenumber of true positive results detected by the models or decreasing thenumber of false positive results detected by the models.
 10. The systemof claim 1, wherein normalization further comprises: converting text toa default character encoding; performing segmentation and entityrecognition; and recognizing at least one of contractions, emoticons,redundant repetitions, whitespaces, tags, markers, and code snippets.11. The system of claim 1, wherein correction further comprises:correcting typos, misspellings, word boundaries, and characterreplacements; converting enumerations and listings; and resolvingsyntactic errors based on verifying part-of-speech agreements.
 12. Thesystem of claim 11, wherein correction comprises one or more approaches,the approaches comprising: a dictionary-based approach comprising usinga plurality of algorithms for evaluating similarity metrics betweenmisspelled words and correct dictionary entries; and a context-basedapproach comprising using the plurality of algorithms for one or more offinding a replacement and preparing a list of candidates forreplacements for other modules and methods, based on establishing whatelement is required or expected within a given context.
 13. The systemof claim 1, wherein transformation further comprises execution of thetransformation module comprising: replacing certain expressions andentities with their equivalents recognizable by the system for detectingone or more predetermined online behaviors; removing irrelevantexpressions and entities; fulfilling omitted expressions and entities;and recognizing quotations and indirect speech.
 14. The system of claim1, wherein the at least one processor is further configured as adetection engine in order to perform online violence detection,including at least one functional module for detecting online violencetargeted against a second person, wherein the second person ischaracterizable as an interlocutor.
 15. The system of claim 14, whereindetecting online violence targeted against the second person comprisesthe use of contextual models for detecting a plurality of types ofonline violence comprising: personal attacks; threats; blackmail; sexualharassment; persuading harmful behaviors; and sexual and childexploitation.
 16. The system of claim 14, wherein the detection enginefurther comprises at least one module performing transformations tosecond person that transforms normalized, corrected, and transformedtext so as to enable detecting online violence targeted against thesecond person to further detect online violence targeted against anyother individual or group.
 17. The system of claim 16, whereinperforming transformations to second person comprises a plurality oftext transformations, comprising replacing third person personalpronouns, named entities that can be objects of online violence, andterms related to minorities and other groups of people, with respondingsecond person references.
 18. The system of claim 14, wherein thedetection engine further comprises at least one module for detectingother online violence that cannot be detected with the at least onefunctional module for detecting online violence targeted against asecond person, wherein detecting comprises one or more of performingtransformations to second person, and not performing transformations tosecond person.
 19. The system of claim 18, wherein detecting otheronline violence comprises using contextual models to detect theplurality of types of online violence comprising: sexting andsex-related content; self-harm declaration; sexism; racism; ageism;ableism; homophobia; and any other hostility to, prejudice against, ordiscrimination against minorities and any other individuals and groups.20. A natural language processing system comprising: at least oneprocessor that receives online data from any sources on the internet andprocesses the online data to determine whether the online dataimplicates online violence, wherein processing the data comprises, theat least one processor receiving plain text input; the at least oneprocessor performing a plurality of functional modules, the plurality offunctional modules comprising one or more of a text preparation module,a normalization module, and a correction and transformation module thatgenerate the normalized text, the corrected text, and transformed text,respectively, wherein normalization comprises the process oftransforming the plain text input into a single predetermined canonicalform to generate normalized text, correction comprises the process ofrevising the normalized text in order to correct misspellings and anyother types of intentional and non-intentional errors to generatecorrected text, and transformation comprises the process of revising thecorrected text in order to replace, remove or add specific textfragments to generate transformed text that is optimally phrased fordetecting one or more predetermined online behaviors; and the at leastone processor performing online violence detection on the normalizedtext, the corrected text, and transformed text, including, at least onemodule detecting online violence, wherein the at least one module iscomprised of a set of contextual models wherein, each contextual modelcomprises a detection model for detecting a defined single type ofonline violence; each contextual model classifies text as containing ornot containing a given type of online violence and assigns a set ofprocessing-related information to the text; and the at least onecontextual model is customizable by a user to alter one or more of thedefinitions of the single type of online violence and how the text isclassified.
 21. The system of claim 20, wherein a contextual model iscustomized by turning off and on one or more detection rules in order todefine which subparts of a given online violence phenomenon should bedetected by the contextual model.
 22. The system of claim 21, whereininformation regarding which detection rules should be turned off and onwithin a community can be obtained by utilizing annotated datasets fromthe community and supervised machine learning methods, by analyzingcommunity guidelines, or by interviewing people responsible for thecommunity.
 23. The system of claim 22, wherein a process of obtainingand applying information regarding which detection rules should beturned off and on can be performed manually, semi-automatically orautomatically.
 24. The system of claim 20, wherein a contextual model iscustomized by altering dictionaries, wherein altering dictionariescomprises changing which words, phrases and expressions should bedesignated as offensive.
 25. The system of claim 24, wherein theoffensive words, phrases and expressions are assigned a severity level.26. The system of claim 24, wherein the dictionaries are changedmanually, semi-automatically or automatically, utilizing supervised andunsupervised machine learning methods.
 27. The system of claim 20,wherein a contextual model is customized by utilizing a TransformationsTo Second Person module, wherein a set of words, phrases and expressionsreferring to a selected individual or group is transformed into secondperson references in order to make the individual or group being treatedby the system as a potential violence object.
 28. The system of claim27, wherein the Transformations To Second Person module can be utilizedto exclude a selected individual or group from being treated by thesystem as a potential violence object.
 29. The system of claim 20,wherein a contextual model is customized by altering combinations of thedictionary entries and detection rules in order to specify how theselected dictionary entries should be treated by the system in differentcontexts represented by the detection rules.
 30. The system of claim 20,wherein a contextual model is customized by re-assigning detection rulesin order to create a new hierarchical structure of violence categories.